The objectives for this lab are to get you to:
Before logging on to the server, we need to make sure your computer is set up correctly. In order to access the CHPC servers and run GRASS you will need an ssh client, an X server and a program that can copy files between your computer and the server.
We are going to use the CHPC’s unallocated server, lonepeak, for this class. The address you will need for this server is: lonepeak.chpc.utah.edu.
If you are using Windows, you will need to install the following bits of software:
These should already be available on the lab computers. On your own computer, you will need to download and install them. Once they are installed, we can set these up.
First start WinSCP to allow transfer of files between your computer (the client) and the server. WinSCP will open with the following screen asking you for login details for a server.
On this screen:
SFTPBefore clicking the [Login] button, click [Save] to save these details so you won’t have to enter them each time. Do NOT click on the ‘Save password’ check box. Save the session as ‘lonepeak’.
Now click login (it may prompt you for your password again),. Click [Yes] if it asks you about the key. It will ask you to enter your password. Do so, and a window will open showing your folders on your computer (left panel) and the server (right panel). You can now use this to copy and move files.
We now need to add some data to start using GRASS. We’re going to use a different file containing the North Carolina dataset (nc_spm_08_grass7.tar.gz) in order to demonstrate some Linux functions. Download this from Canvas, and find the file by browsing through the left-hand window, and drag it across to the right hand panel.
Next, start Xming from the programs menu. This will not open any windows, but instead launch a server. To check that it has started, an icon should appear in the tool tray (in the bottom right hand corner of the screen) - you should see a small ‘X’ icon:
Now start putty from the program menu. This will open a window with a large number of options. Fortunately, we only need to change a couple of these. In the first screen (the session window), add the name of the server to the ‘Host Name’ window, and make sure the ‘Port’ is set to 22 and the ‘Connection type’ is set to ‘SSH’.
We need to set the connection so that it accepts “X forwarding”, in other words, allows graphical windows from the server to be opened on the client computer. So go to the ‘SSH’ option on the left hand side, then the ‘X11’ option. On this window, check the tickbox labelled ‘Enable X11 forwarding’.
Now return to the session window (left hand panel). As we are going to be using this several times during the semester, we will save the session details. Enter a name in the box ‘Saved sessions’, then click on [Save]. Now highlight the saved session name, and click on [Open] to start the session. Once the connection is established, putty will open a terminal and ask you to login. Enter here your uID and password when prompted. If prompted to save the SSH key, press yes.
If you’ve done all of this, you’ll see a welcome screen, and a prompt:
This terminal opens automatically a bash shell, which allows you enter commands and run programs (we will go over this in a couple of weeks). For now, just try running the following commands:
pwd
which prints the name of your home directory (where the terminal starts) and
ls
which shows a list of the files in your home directory. Make sure that you can see the compressed file with the North Carolina dataset.
You can logout of the terminal by simply typing:
logout
Once you have Cyberduck installed, open it from the Applications menu. The main page shows all your saved sites and allows you add new ones. We’ll do this now for the CHPC server. Click on [Open connection] in the top left of this window. This will bring up the following menu:
Here, make sure that the connection protocol is set to SFTP, then enter the server name and your uID and password. Click [Connect], and a window will open showing your home directory.
We now need to add some data to start using GRASS. We’re going to use a different file containing the North Carolina dataset (nc_spm_08_grass7.tar.gz) in order to demonstrate some Linux functions. Download this from Canvas, and find the file by browsing through the left-hand window, and drag it across to the right hand panel.
If you want to be able to view any of GRASS’s graphics windows you will also need a X11 client. Mac OSX systems do come with a built in client, but this has been deprecated by Apple, and they recommend installing XQuartz. If you don’t already have this on your computer, download it and install it now. Once installed, this should start automatically when you log in to the server by ssh.
To login to the server on a Mac, simply open a terminal and enter the following command (replacing uID with your uID):
ssh -Y uID@lonepeak.chpc.utah.edu
And you will be prompted to enter your password. The flag -Y enables X-forwarding, which will allow you to open graphics windows. You may get a question about the key-pairs; just click ‘Yes’.
This terminal opens automatically a bash shell, which allows you enter commands and run programs (we will go over this in a couple of weeks). For now, just try running the following commands:
pwd
which prints the name of your home directory (where the terminal starts) and
ls
which shows a list of the files in your home directory. Make sure that you can see the compressed file with the North Carolina dataset.
You can logout of the terminal by simply typing:
logout
If you are already working in Linux, then there is no additional software to install.
The easiest way to copy files is to connect using the file browser (these instructions are for Ubuntu). Open Nautilus, then go to the [File menu] > [Connect to server]. This should open the following window.
Here, set - Server to lonepeak.chpc.utah.edu - Type to SSH - Enter your uID in the User name, and enter your password.
Click [Connect] to open your Home directory CHECK
To login to the server using Linux, simply open a terminal and enter the following command (replacing uID with your uID):
ssh -Y uID@lonepeak.chpc.utah.edu
And you will be prompted to enter your password. The flag -Y enables X-forwarding, which will allow you to open graphics windows. You may get a question about the key-pairs; just click ‘Yes’.
This terminal opens automatically a bash shell, which allows you enter commands and run programs (we will go over this in a couple of weeks). For now, just try running the following commands:
pwd
which prints the name of your home directory (where the terminal starts) and
ls
which shows a list of the files in your home directory. Make sure that you can see the compressed file with the North Carolina dataset.
You can logout of the terminal by simply typing:
logout
Like the family of UNIX-like operating systems before it, the primary interface to Linux is the command-line. The command-line is still the most powerful and direct way to interact with the system. Command-line interfaces are implemented as shells in Linux. Each shell has varying capabilities and features and the user should choose the shell that best suits their needs. The shell is simply an application running on top of the kernel and provides a powerful interface to the system. The default shell for the CHPC servers is BASH.
The shell is your workspace in the terminal. It is a command language interpreter that executes commands read from the standard input device or from a file.
| Shell Name | Developed by | Where | Remark |
|---|---|---|---|
| BSH (Bourne-Again SHell) | Brian Fox and Chet Ramey | Free Software Foundation | Most common shell in Linux |
| CSH (C SHell) | Bill Joy | University of California (For BSD) | The C shells syntax and usage are very similar to the C programming language |
| KSH (Korn SHell) | David Korn | AT & T Bell Labs | |
| TCHS | See the man page | Enhanced version of CSH |
To find available shells in your system type:
cat /etc/shells
cat types out the contents of a file as you saw above. The file “shells” in the “/ etc” directory holds all available shells in the system.
To find your current shell type:
echo $SHELL
The command echo echos or types what follows it. The $ sign says echo the contents of the environmental variable SHELL. This is a system variable set up in your init file or by using the export command.
The shell’s environment can be defined as
a set of persistent knowledge under which the shell operates.
This information is stored as “environmental variables”. These are either global variables that are set for the entire system, or user variables that can be customized by the user using a specific set of files contained in their HOME directory. Note that when you open a session, your environment is automatically setup using both the global and user settings.
Being able to know which environment variables affect which aspect of your environment means that you are able to customize your environment. In many shell scripts and using many applications or programming languages you will need to know the values of these variables. Some useful variables are:
PATH: the variable that allows us to locate a command. Which gives all of the directories that the system should search through when looking for a specific command. To see what is in your path:
echo $PATH
PWD which holds the path of the current working directory:
echo $PWD
HOME which holds the user’s home directory:
echo $HOME
USER holds the user name of the current account (note that you can get the same information by typing whoami):
echo $USER
To see the overall value of your environment:
env
This last command produces quite a lot of output, more than can be seen in the terminal. A very useful command when working in the terminal is less. This displays only the amount of output that will fit in a single terminal screen. Here we use the Unix pipe operator | to pipe the output of the env command to less. You can now scroll through this with the arrow keys, or page through it using the space bar. Pressing ‘Q’ at any point will quit the less function.
UNIX uses a true hierarchical file system, which is very different from the Windows structure. The UNIX file system starts at a single point: / (pronounced “root” [not to be confused with the root user or /root, the root user’s home directory]). Components
The UNIX file system is comprised of several parts:
One of the most difficult thing to grasp about the UNIX file system is that everything in UNIX is treated as a file. This includes directories, special files, partitions, disk drives, mice, serial ports, printers, displays, network interfaces, etc.
Open the Terminal window and navigate to the /dev directory using the change directory (cd) command:
cd /dev
Then list the files in this directory:
ls
Note that the list includes USB ports, CD/DVD-drives, etc.
The UNIX file system is analogous to a tree. The root, /, is the root of the hypothetical tree and directories are the branches. The UNIX file system can have other file systems attached onto them, and may be comprised of many different file systems. The mount command is used to attach other file systems or to show existing attachments.
The UNIX file system is traditionally divided into a number of common directories. The series of directories, separated by a slash (“/”) and often ending with a file, is called a path. Each directory path has a specific function. Below is a table listing some of the common directories and their normal use:
| Path | Windows equivalent | Description |
|---|---|---|
| / | C: | The start or ‘root’ of the file system |
| /home | My Documents or Profiles | User’s private files |
| /usr or /usr/local | Program Files | Installed software |
| /usr/lib or /usr/local/lib | Windows\System | Shared libraries |
| /dev | Windows\System | Device drivers |
| /etc | The Registry or .ini files | Application and operating system configuration information |
| /tmp | Windows\Temp | Temporary system files |
| /bin or /sbin | Windows | System executable files |
In the Terminal window, type
cd /
and press enter/return to “change directory” to the “root” directory.
This is the directory that all other directories spring from. Which, as in a tree is why it is called “root”. In the Terminal window, type
ls
to see the contents.
Now go to the /home directory and list all the current users accounts (directories - note this won’t work on the server)
cd /home
ls
Now return to your home directory and list the files there:
cd
ls
Note that adding the parameter -al provides more information about the files
ls -al
The following are some of the more frequently used file system commands:
| Command | Description | DOS/Windows Equivalent |
|---|---|---|
| awk | File processing and report generating | N/A |
| cat | Show contents of a file | type or double-clicking a file |
| cd | Change directory | cd or double-clicking a folder |
| cp | Copy a file | copy or dragging a file |
| emacs | Edit a file | edit or notepad |
| file | Determine file type | file extension or right-click properties |
| find | Find a file | Windows Explorer Find |
| grep | Find lines in a file | N/A |
| gzip | File compresser | WinZip/7Zip |
| less | Display a file one page at a time (extended version of more) | more |
| ln | Link a file to another file | Create Shortcut |
| ls | Display files in a directory | dir or Windows Explorer |
| mkdir | Create a directory | mkdir or creating a folder |
| more | Display a file one page at a time | more |
| mv | Move or rename a file | rename or dragging a file |
| rm | Remove a file | delete or deleting a file |
| rmdir | Remove a (empty) directory | rmdir or deleting a folder |
| sed | Stream editor | N/A |
| tar | Archive creator | N/A |
| unzip | File compressor | WinZip/7Zip |
| vim | Edit a file | edit or notepad |
| wc | Count words, lines and characters in a file | N/A |
For this part, you will need the file nc_spm_08_grass7.tar.gz from Canvas. Download this file, and copy it to your home directory on the server (if you haven’t already)
Now log in to the server using XMing and putty (or terminal ans ssh). First up, we’ll need to create a GRASS database directory, which we can do with mkdir.
mkdir grassdata
Now move the compressed data file to this directory:
mv nc_spm_08_grass7.tar.gz grassdata/
Now change to this directory, and check that you have copied the file to the right place:
cd grassdata
ls -l *.tar.gz
The file is a gzipped archive file (these have the extension .tgz or .tar.gz). To open these and extract the files, we need two programs: gunzip and tar. We start by decompressing the file using gunzip
gunzip nc_spm_08_grass7.tar.gz
ls -l *.tar
You should see that the .gz extension has been removed, and the size has increased. Now we can list the contents of the archive file:
tar tvf nc_spm_08_grass7.tar
The parameter tvf tells tar to simply list the contents rather than extracting. Remember that the terminal allows tab completion. So rather than typing the entire file name, try typing ‘nc_’ and pressing ‘TAB’, and the file name should complete. Now we extract the contents, simply by changing the t to an x:
tar xvf nc_spm_08_grass7.tar
ls
You can combine these two steps into one: tar zxvf both decompresses the archive and extracts the contents. Note also that Linux has an unzip command that will work with standard compressed zip files, but as the tar/gzip format is relatively common, it is worth seeing how it works. Change directory to the new directory and list the contents:
cd nc_spm_08_grass7
ls -al
You should see a set of directories, and some files. The first column gives the file permissions (more on this later). If the first letter is a d then this is a directory. Alternatively, type ls -alG and the directories will be shown in a different color.
Now look at the contents of a file:
cat VERSION.txt
In the Terminal window type
cp VERSION.txt myfile1
to copy the version file you just looked at, making a second identical file with a new name, myfile1.
In the Terminal window type
file myfile1
to determine the file type.
To create a new directory called “mydir” type:
mkdir mydir
To copy the readme file into mydir keeping the name “readme” type:
cp VERSION.txt mydir/
To copy the readme file into mydir with another name “myreadme” type:
cp VERSION.txt mydir/myVERSION.txt
To list all the files in the new directory “mydir” type:
ls -l mydir
To copy an entire directory (e.g. landsat/) to the directory “mydir” with the same name type:
cp -r landsat mydir/.
To find the number of newlines, words and bytes in a file type:
wc VERSION.txt
To rename a file type:
mv VERSION.txt myVERSION.txt
Copy again so you can remove:
cp myVERSION.txt VERSION.txt
To remove a file type:
rm myVERSION.txt
In Linux a user is identified by their username. Linux usernames are similar to Windows logon IDs. Linux maintains a list of users, called a name space, either locally on each individual machine or across the network via NIS for a group of machines.
User == uid
Each user on a Linux machine has a unique username that corresponds to a unique uid. The uid is a numeric value used to determine if the user is authorized to perform a particular task. It is important to note that the Linux operating system is concerned only with the numeric value of the uid and not the text that describes the user’s name.
Group == gid
Groups are also associated with unique numbers called gids. Users may belong to more than one group, although they can be associated with only one group at a time. Groups may have zero, one, or more members. Like uids, the Linux operating system is concerned only with the numeric value of the gid and not the textual name of the group.
Each time a user tries to login, Linux attempts to authenticate that user against a name space. The login process prompts for a username and, usually, a password. Linux then encrypts the entered password and checks to see if the encrypted text matches the stored version of the user’s encrypted password. If there is a match, the user is authenticated; if not, the user is denied access.
Since Linux treats everything as a file, learning about file permissions is very important. Linux allows access to a file based upon affiliation as defined by user, group, and other. The possible permissions for each affiliation are read (r), write (w), and execute (x). Linux actually uses one bit to indicate these three permissions for each of the three affiliations.
In the Terminal window, type
ls -l /etc/passwd
Your output should look similar to this:
-rw-r--r-- 1 root root 4429 May 1 2019 /etc/passwd
The first dash is a flag specifying the type of file. The next three characters are the permissions for the user who owns the file (root). The next three characters are the permissions for the group associated with the file. The final three characters are the permissions for any user who is not the owner of the file nor belongs to the group associated with the file.
Each of the permissions has a different meaning depending on whether it is set for a file or a directory:
| Type | r |
w |
x |
|---|---|---|---|
| File | Can read the file | Can change the contents or delete the file | Can execute the file |
| Directory | Can see a listing of files in the directory | Can create new files in the directory or delete files from the directory | Can change into that directory |
In the case of the /etc/passwd file, only the account root (the superuser) can write to the file, but everyone can read it.
We will now try some examples of shell scripting using the BASH shell. This is the easiest way to automate tasks in Unix, including analyses in GRASS. You will need a text editor to write the scripts, and there are several that can be used, including:
Here we will use vim, a standard text editor that can be used through the terminal. To start this with a new script, type:
vim myscript.sh
A window will open that looks like this:
More detail on the vim editor is given below, but for now, press i to enter text insertion mode, then type the following lines
#!/bin/bash
clear
echo "Hello World - this is my first shell script"
Now hit the ESC button to leave text insertion mode, and type :wq to save your input and quit the editor. After writing your scripts you need to change the permissions, so the execute permission is on for the user, the the script can be run from the command line by prefacing it with ./:
chmod u+x myscript
./myscript.sh
Lets create another script to print user information, who is currently logged in , current date & time:
vim ginfo.sh
Enter the following text:
#!/bin/bash
clear
echo "Hello $USER"
echo "Today is \c ";date
echo "Number of user login : \c" ; who | wc -l
echo "Calendar"
cal
exit 0
Now, exit the file as before (ESC then :wq), set the permissions, then execute the script.
One of the things that are very important is to see some of the information in a given file. Make sure you are in the ‘nc_spm_08’ directory, and we will look at the file HISTORY.txt. Commands to do this include:
cat - shows you the contents of an entire file (you will see only the last screen full of lines).
cat HISTORY.txt
less - shows you the contents of an entire file one screen at a time:
less HISTORY.txt
head - shows you the first 10 lines of a file
head HISTORY.txt
to see fewer or more lines us the -n option
head -3 HISTORY.txt
tail - shows you the last 10 lines of a file
tail HISTORY.txt
tail -3 HISTORY.txt
Most command line programs will take some form of input, and give some form of output. By default, this is either taken from STDIN and written to STDOUT (i.e. the terminal). Redirection allows you to use files instead for this, using the > and < symbols. For example, the following takes the output of the ls function and writes it to a file
ls /etc > filelist.txt
head filelist.txt
The next example uses the cat command, and takes input from the file VERSION.txt in the ‘nc_spm_08’ directory]
cat < VERSION.txt
The other part of redirection uses pipes to pass output from one program to a second. Using pipes is extremely useful in Linux, as it allows you to combine several simple command line programs to make a fairly complex set of file and data manipulation. A pipe between two programs is given by the | symbol. For example, this lists the files in the directory, then passes this list through the sort function with a -r parameter to do a reverse sort:
ls | sort -r
Here the pipe has the line count command (wc) read its input from the ls command’s output. The result is the number of files in the directory.
ls | wc -l
And here we pipe the output of ls through grep to select only the lines with the character string ‘li’ in them
ls | grep li
Vi and vim are text editors (vim is an ‘improved’ version of vi). Vi comes on every UNIX-like system by default, and vim is usually available. While there are many other text editors, and different people believe different ones are the best, we will look at vi/vim here as it can always be found on a UNIX-like OS, unless the system administrator has removed it (unlikely). We will concentrate on vim as it is a little easier to use, but the basic commands can be used in vi, if this is all that is available.
Vi/vim has two modes:
To create a file, choose a name that doesn’t exist in the directory, e.g. vim newfile. This creates the new file, opens it and puts you in command mode. Lets create a new file, a list of files for the North Carolina dataset.
ls > filelist.txt
vim filelist.txt
The file opens in command mode. If the file doesn’t exist then you will see a screen with tildes “~”. If the file does exist you will see the first page of text in the file.
Basic movement commands. You must be in command mode to move around to get into command mode press the escape key . Once you are in command mode you can use the arrow keys and page-up and page-down to move around. Other useful commands for movement:
H - moves you to the first character of the “highest” line on the screenM - moves you to the first character at the “middle” of the screenL - moves you to the first character at the “bottom” of the screen$ - moves you to the end of the current line^ - moves you to the beginning of the current linef moves you one page toward the beginning of the fileb move you one page toward the end of the file:, followed by a line number to go to that lineIn order to add, delete or change text, we need to change to text mode. In order to enter text mode, you need to use one of the following commands:
i - insert new text to the left of the character the cursor is onI - insert new text to the left of the first non-blank character on the linea - insert new text to the right of the character the cursor is onA - insert new text to the right of the last character on the lineo - inserts a new line below the line the cursor is onO - inserts a new line above the line the cursor is onNow that we know the basic commands, we can make some changes to this file:
O key to insert a line above the existing text:wqTyping :wq saves and closes the file. The : tells vi that a command is coming up next, the w tells vi to write the file to disk (save the file) and the q tells vi to quit. If you don’t type the q vi saves the file and lets you continue. If you have made changes that you don’t want to keep, then typing :q! will quit without saving.
The three main commands for copying and pasting are:
d - cut texty - copy textp - paste textReopen the file you have been using in vim, and try the following
d. Nothing will happen until you press the direction key. Up or down will cut the line above or below the cursor, and left or right will cut the first character in that direction. Press right to cut the character under the cursorp to paste the text to the right of where the cursor is. Pressing shift-p will paste to the leftd in the first step, press y to copy5y, then go to the end and press p. The first five characters will be copied and pasteddd. The line will be cut, then move to where you want the line and press p to paste. As before, entering a number before the dd will cut multiple lines.Vi/vim has a quite sophisticated (and pretty complex) find and replace method. To find anything in the file, simply use / while in command mode. For example /a will find all the ’a’s in the file. Press n to move to the next one.
Find and replace is done using the :s command. This takes two arguments, the character string to search for, and the character string to put in its place. Try the following
ESC) and type :s/a/u. Press return and the first ‘a’ will be changed to a ‘u’.:s/E/D/g. Appending the “g” onto the end now changes all the first character to the second (NB you are not restricted to single characters here).% between the : and the s. For example: :%s/txt/csv/g will change every ‘txt’ to a ‘csv’:1,3s/csv/txt/g to change the string ‘csv’ to ‘txt’ on the first three lines onlyThere is much more that can be done, including the use of regular expressions. A good site for examples and tutorials is the vim site.
man command-name. Type:man cp
This gives the information on the cp (copy) command. Press ‘q’ to quit.
Other options:
whatis gives a one-line description of the command but doesn’t include the options: whatis cpapropos gives a list of commands with this keyword in their description: apropos copyOpen source software is generally tightly linked to the development of the internet, and the greatest source of help, tutorials and examples can be found there. Particularly recommended are
Finally, we’ll start GRASS and run some simple commands. Make sure that you have set up the grassdata directory and unzipped/untarred the data files. Now at the command line type:
grass78
Assuming that you have correctly set up X forwarding, this will open the usual GRASS project and location menu:
Select the nc_spm_08_grass7 location and user mapset (or create your own mapset), and click [Start Grass session]. Your GRASS session should now be running in the console.
From here, you can run all the usual GRASS commands. Try a few of these now:
g.list rastr.info elevationr.univar -e elevationr.report lakes units=a,cIf X is running correctly, we can now open a display window:
d.mon start=wx0
d.rast elevation
Find and show all elevations over 100m
r.mapcalc "tmp = if(elevation >= 100)"
d.rast tmp
When you are done, be sure to stop all open display monitors
d.mon stop=wx0
And close GRASS
exit
Now enter logout on the command line to close your session on the CHPC.
Simon Brewer 01/21/14. Updated for CHPC access on 02/20/20